Watch-time clustering for improving video searches, selection and provision

ABSTRACT

This document describes, among other things, systems, methods, devices, and other techniques for using information about how long various videos were presented at client devices to determine subsequent video recommendations and search results. In some implementations, a computing can include a modeling apparatus, a front-end server, a request manager, one or more video file storage devices, a video selector, or a combination of some or all of these. The video selector can select video content for a particular digitized video among a plurality of digitized videos to serve to a computing device responsive to a request. The selection can be based at least in part on how long the particular digitized video has been presented at client devices associated with users having characteristics that match one or more characteristics of the user that submitted the request for video content, as indicated by the modeling apparatus.

TECHNICAL FIELD

The subject matter of this document generally relates tocomputer-implemented techniques for searching digitized video contentand determining video content to serve over a communications network.

BACKGROUND

Video-sharing services provide a platform for video content creators todistribute digitized videos to other parties over a network, such as theInternet. The video-sharing service may be implemented by a computingsystem comprising one or more servers in one or more locations. Contentcreators can upload their videos to the computing system, along withmetadata that describes their videos, which the computing system uses toindex the videos and make them discoverable to users who express aninterest in viewing such videos. The video-sharing service can thenreceive search queries from users requesting video content. In someinstances, the video-sharing service selects videos to serve to usersbased on a comparison of topics associated with the videos to topicsindicated in search queries that the users have submitted to thevideo-sharing service.

SUMMARY

This document generally describes in some aspects systems, methods,devices, and other techniques for using information about how longvarious categories of users have watched online videos to improve videosearch results and recommendations. When a requesting user is providedwith a set of video search results that does not include videos that therequesting user was expecting to receive for a particular query, theuser may revise the query in an iterative manner until more suitablevideos are identified for the requesting user. Sometimes, even if theresults do include one or more videos that the user expected to receive,they may not be among the top results that are displayed to a user. Thiscan result in an unnecessary usage of valuable network resources, i.e.use of processing resources in servers, client or routers, bandwidth oncommunication links etc., simply for processing multiple search requestssubmitted by the requesting user. Such searching techniques areconsidered inefficient and are a drain on valuable processing andnetwork resources. The present embodiments aim to overcome theseinefficiencies. The embodiments described herein propose making use ofone or more measured and/or stored attributes and characteristicsconcerning the users or the user devices that are requesting the videos,or that have requested the videos in the past to provide a set of searchresults to the requesting user. For example, a video search system mayuse watch-time information as a heuristic for ranking and selectingvideo content to serve to users. In some implementations, the videosearch system may further score video content creators based on how longtheir videos have been watched by various categories of users, and thecreators' scores can be used as a heuristic for ranking and selectingvideo content to serve to users.

In some implementations, the subject matter described herein can includea computing system for serving videos over a network. The systemcomputing can include a modeling apparatus, a front-end server, arequest manager, one or more video file storage devices, a videoselector, or a combination of some or all of these. The modelingapparatus (i) obtains, for various digitized videos that the computingsystem has presented at client devices over a network, watch timeinformation that specifies an amount of time that the various digitizedvideos were presented at the client devices, (ii) groups the watch timeinformation into different groups based on characteristics of usersassociated with the client devices, and (iii) determines, based on thegrouping of watch time information, how long the various digitizedvideos were presented to different groups of users. The front-end serverreceives a request for video content and serves video content identifiedin response to the request over the network to a computing device thatis separate from the computing system. The request manager, includingone or more processors, analyzes the request for video content andidentifies selection criteria in the request, including identifying oneor more characteristics of a user that submitted the request for videocontent. The one or more video file storage devices store a plurality ofdigitized videos that have been made available by various parties fordistribution over the network. The video selector, including one or moreprocessors, selects, from the video file storage devices, video contentfor a particular digitized video among the plurality of digitized videosto serve to the computing device in response to the request, wherein theselection is based at least in part on how long the particular digitizedvideo has been presented at client devices associated with users havingcharacteristics that match the one or more characteristics of the userthat submitted the request for video content, as indicated by themodeling apparatus.

These and other implementations can optionally include one or more ofthe following features.

The modeling apparatus can group the watch time information intodifferent groups by assigning respective portions of the watch timeinformation to different viewer categories that correspond to differentsets of characteristics of the users associated with the client devicesat which the various digitized videos were presented. The modelingapparatus can further identify, for each respective viewer category, adistribution of watch time of users in the respective viewer categoryamong digitized videos in a group of digitized videos that were at leastpartially presented at computing devices associated with users in therespective category.

For each respective digitized video among the plurality of digitizedvideos stored on the one or more video file storage devices, themodeling apparatus can: identify, from the watch time information andfor each respective viewer category among a plurality of viewercategories, how long the respective digitized video was presented atclient devices associated with users in the respective viewer category;and assign to the respective digitized video, for each respective viewercategory among the plurality of viewer categories and based on how longthe respective digitized video was presented at client devicesassociated with users in the respective viewer category, a score thatindicates a relevance of the respective digitized video to therespective viewer category.

The video content for the particular digitized video selected by thevideo selector can include at least one of the particular digitizedvideo, a title of the particular digitized video, a description of theparticular digitized video, or a hyperlink to the particular digitizedvideo.

The one or more characteristics of the user that submitted the requestfor video content can include at least one of demographiccharacteristics of the user or behavioral characteristics of the user.

The one or more characteristics of the user that submitted the requestfor video content can include demographic characteristics that indicateat least an age or gender of the user.

The one or more characteristics of the user that submitted the requestfor video content can include behavioral characteristics that indicateone or more queries submitted by the user in a same session in which theuser submitted the request for video content. The video selector canselect to serve video content for the particular digitized video amongthe plurality of digitized videos based at least on how long theparticular digitized video was presented at client devices associatedwith other users during sessions in which the other users also submittedthe one or more queries or at least one query determined to be similarto the one or more queries.

The one or more characteristics of the user that submitted the requestfor video content can include behavioral characteristics that indicate anavigation history of the user associated with the request for videocontent. The video selector selects to serve video content for theparticular digitized video among the plurality of digitized videos basedat least on how long the particular digitized video was presented atclient devices associated with other users having a same or similarnavigation history as the user that submitted the request, in browsingsessions that led the other users to watch the particular digitizedvideo.

The video selector is further configured to select video content toserve to various computing devices as recommendations to respectiveusers of the various computing devices.

In response to receiving the request for video content, the requestmanager can be configured to identify one or more topics of therequested video content, including identifying the topics from at leastone of a user-entered query that is a part of the request or identifyingthe topics from information associated with the user that is notincluded in the request. The video selector can be configured to selectvideo content to serve to the computing device by: identifying digitizedvideos that are determined to be relevant to the one or more topics;ranking the identified digitized videos based at least on how long eachof the identified digitized videos has been presented at client devicesassociated with users having characteristics that match the one or morecharacteristics of the user that submitted the request for videocontent; and based on the ranking, selecting video content for one ormore top-ranked digitized videos to serve to the computing device,including the video content for the particular digitized video.

The system can further include an instrumentation manager, including oneor more processors, that generates and embeds computer code in variousweb pages that the computing system serves to various client devices forthe presentation of various digitized videos, wherein the computer codeis configured, when executed on the various client devices, to cause thevarious computing devices to periodically report over the network and tothe computing system amounts of time that the various digitized videoswere presented at the various client devices.

Some implementations of the subject matter described herein can includea computer-implemented method. The method can include obtaining at acomputing system data that indicates, for each respective computingdevice among a plurality of computing devices to which at least aportion of a first video hosted by the computing system was served, arespective watch time for the first video that occurred at therespective computing device; for each of the respective watch times,correlating the respective watch time with one or more viewer categoriesamong a plurality of viewer categories based on one or morecharacteristics of a user of the respective computing device at whichthe respective watch time occurred; for each respective viewer categoryamong the plurality of viewer categories, accumulating the respectivewatch times that are correlated with the respective viewer category togenerate an accumulated watch time for the respective viewer category;and after the correlating, determining whether to provide video contentassociated with the first video to a first user based at least on theaccumulated watch time of the first video for a viewer category thatmatches the first user.

These and other implementations can optionally include one or more ofthe following features.

Determining whether to provide video content associated with the firstvideo to the first user can include determining whether the accumulatedwatch time of the first video for the viewer category that matches thefirst user meets a threshold watch time.

Determining whether to provide video content associated with the firstvideo to the first user can include: ranking a plurality of candidatevideos, including the first video, based at least on respectiveaccumulated watch times of the plurality of candidate videos for theviewer category that matches the first user; and selecting video contentassociated with one or more videos of the plurality of candidate videosbased on the ranking.

The plurality of viewer categories can classify viewers based on atleast one of demographic characteristics of viewers and onlinebehavioral characteristics of viewers.

For each respective viewer category among the plurality of viewercategories, a respective quality score can be assigned to the respectiveviewer category;

A first watch time of the first video by a user in a first viewercategory among the plurality of viewer categories can be identified; inresponse to identifying the first watch time, a performance score of acreator of the first video can be increased by an amount that is basedon the first watch time and the respective quality score assigned to thefirst viewer category; a second watch time of the first video by a userin a second viewer category among the plurality of viewer categories canbe identified;

In response to identifying the second watch time, the performance scoreof the creator of the first video can be increased by an amount that isbased on the second watch time and the respective quality score assignedto the second viewer category.

The method can include determining the respective quality scoresassigned to respective viewer categories based on revenue generated as aresult of viewers in the respective viewer categories watching videoshosted by the computing system.

Some implementations of the subject matter described herein can includeone or more non-transitory computer-readable media having instructionsstored thereon that, when executed by one or more processors, causeperformance of operations. The operations can include obtaining at acomputing system data that indicates, for each respective computingdevice among a plurality of computing devices to which at least aportion of a first video hosted by the computing system was served, arespective watch time for the first video that occurred at therespective computing device; for each of the respective watch times,correlating the respective watch time with one or more viewer categoriesamong a plurality of viewer categories based on one or morecharacteristics of a user of the respective computing device at whichthe respective watch time occurred; for each respective viewer categoryamong the plurality of viewer categories, accumulating the respectivewatch times that are correlated with the respective viewer category togenerate an accumulated watch time for the respective viewer category;and after the correlating, determining whether to provide video contentassociated with the first video to a first user based at least on theaccumulated watch time of the first video for a viewer category thatmatches the first user.

Some implementations of the subject matter described herein can includea computing system for serving videos over a network. The computingsystem can include: a modeling apparatus that obtains watch timeinformation for various digitized videos that the computing system haspresented over a network, the apparatus including a first processor formeasuring one or more attributes associated with one or more users ofthe system, including measuring watch time information of one or morevideos by a respective user (e.g., measuring how long the one or morevideos were presented at a computing device associated with therespective user); the modelling apparatus further configured to groupthe measured watch time information into different groups based onstored or measured characteristics of users that watched the variousvideos, and to determine, based on the grouping, how long differentgroups of users have viewed the various videos; a front-end server thatreceives a request for video content and serves video content identifiedin response to the request over the network to a computing device thatis separate from the computing system; a request manager, including oneor more processors, that analyzes the request for video content andidentifies selection criteria in the request, including identifying oneor more characteristics of a user that submitted the request for videocontent; one or more video file storage devices that store a pluralityof digitized videos that have been made available by various parties fordistribution over the network; and a video selector, including one ormore processors, that selects, from the video file storage devices,video content for a particular digitized video among the plurality ofdigitized videos to serve to the computing device in response to therequest, wherein the selection is based at least in part on how long theparticular digitized video has been viewed by users havingcharacteristics that match the one or more characteristics of the userthat submitted the request for video content, as indicated by themodeling apparatus.

In some implementations, the techniques described herein may realize, incertain instances, one or more of the following advantages. Generally,the watch-time and creator performance heuristics may allow the videosearch system to better identify videos that users are interested inviewing. Similarly, the video search system may reduce occurrences ofserving videos to users' computing devices that are not sufficientlyrelevant to users' search queries, or which the users are otherwise notparticularly interested in viewing. As a result of making better videoselections for users, the search system may conserve server resourcesthat would otherwise be wasted in serving videos to users that the usersare not interested in watching. Similarly, these techniques may conservenetwork bandwidth by reducing the amount of video traffic related topoor video selections, and may conserve power on users' devices byreducing playback time of videos that users are not actually interestedin viewing. In some implementations, the creator performance scores canincentivize creators to generate quality video content that is targetedtoward certain categories of viewers whose interactions with thevideo-sharing service disproportionately enhance the value of theservice.

DESCRIPTION OF DRAWINGS

FIGS. 1A and 1B illustrate conceptual diagrams of various usersinteracting with a video search system to request online video content.In FIG. 1B, the video search system uses watch-time and creatorperformance heuristics to improve the selection of video content forusers.

FIG. 2 depicts a block diagram of an example environment in which videocontent can be captured, analyzed, and shared between computer systems.

FIG. 3 shows a block diagram of an example computing system, including avideo search system and a video storage system that use video watch-timeinformation to determine video content to serve to various computingdevices of viewers in communication with the computing system.

FIG. 4 depicts a flowchart of an example process for generating awatch-time model that a video search system uses to affect the selectionof video content to serve to a user based on historical informationabout how long various groups of viewers have watched different videos.

FIG. 5 is a flowchart of an example process for selecting video contentto serve to a computing device of a viewer using information about howlong other viewers, who are similar to the first viewer, have watchedvarious videos during a past period of time.

FIG. 6 is a flowchart of an example process for determining and usingcreator performance scores to rank video content and to allocate assetsdistributed to creators for improving the quality of videos submitted toa video-sharing service.

FIG. 7 shows an example of a computing device and a mobile computingdevice that may be used to implement the computer-implemented methodsand other techniques described herein.

Like references among the various drawings indicate like elements.

DETAILED DESCRIPTION

Some techniques for providing video sharing services that may provideresults based on query information experience drawbacks as explainedabove, resulting in a strain on useful network and processing resources.The described embodiments herein propose a method of clustering andcategorizing video data based on one or more attributes that are relatedto actions that have been performed by one or more users of the systemor related to characteristics pertaining to said users that are storedon the system. User(s) actions may be past or real-time events oroccurrences, such as data obtained relating to past searches, previousselections, responses to one or more similar videos, etc.Characteristics such as stored details of user preferences and personalinformation may also be used. This document generally describes in someaspects systems, methods, devices, and other techniques for usinginformation about how long various categories of users have watchedonline videos to improve video search results and recommendations.

In some examples, a computing system may implement an onlinevideo-sharing service that hosts videos submitted by a collection ofindependent creators and makes the creators' videos available fordistribution over a network (e.g., the Internet) to one or more viewers.Viewers may access the video-sharing service on respective clientdevices and search videos that that are hosted on the service. Thesystem may also present to viewers recommendations likely to be ofinterest to the viewers.

According to the techniques described herein, historical videowatch-time data may be used as a heuristic for determining video searchresults and recommendations to present to users. For example, thecomputing system may identify demographic or behavioral characteristicsassociated with a particular viewer that has submitted a video searchrequest. Based on the viewer's characteristics, the system may classifythe viewer into one or more viewer categories and determine videos thathave accumulated high watch times from other viewers within the same orrelated categories. Videos with higher watch times within thesecategories may then be promoted, for example, in search results that thesystem returns in response to the viewer's query. In someimplementations, the techniques described herein may additionally oralternatively be applied to score and rank creators that are associatedwith the service based on watch-time metrics for their respective videosin different viewer categories. The system can then use the creators'scores or rankings as a further heuristic for determining video searchresults and recommendations and for distributing assets among thecreators.

Turning to FIGS. 1A and 1B, conceptual diagrams are shown of usersinteracting with a video search system to request online video content.These diagrams show that, in some implementations, a video search systemcan improve selection of video content to serve to a user based onwatch-time heuristics, creator performance heuristics, or both, so as tomore efficiently provide to a user video content that the user is likelyto be interested in. By initially serving more relevant video content toa user, the techniques discussed herein may make better use of systemresources because the user need not make as many queries or selectionsto play different videos before arriving at a satisfactory video. Assuch, the servers of the video search system may use less computationalresources and reduce the expense in identifying and transmittingunsatisfactory video content to the user. Moreover, the bandwidth of acommunications network (e.g., a local area network, wide area network,wireless network, and/or the Internet) that connects the user's deviceto the servers of the video search system may be conserved by reducingthe amount of data for unsatisfactory video content that the networkcarries to user devices. In some cases, this can be especiallysignificant for wireless and cellular data networks in which the amountof bandwidth consumed by individual users is closely regulated so as toensure that, on average, the capacity of the network is not overwhelmedby the activities of a population of users. Streaming video and othervideo content requires a relatively large amount of bandwidth (e.g., ascompared to text, still images, or some other content types), and soreducing the amount of unnecessary video content transmitted on thenetwork can have a significant impact on network performance.

In some implementations, the techniques described herein for usingwatch-time heuristics, creator performance heuristics, or a combinationthereof, to better target video content to users can also improveaspects of users' computing devices. Downloading, streaming, and playingvideo on users' devices are generally computationally expensiveactivities. Some of this expense may be reduced by the presentlydescribed embodiments as the embodiments propose downloading, streaming,and/or playing video content that are identified as ones that the usersare interested in viewing. Moreover, in the context of mobile devicesand other battery-powered computing devices, saving computationalexpense from activities related to obtaining and processingunsatisfactory video content can conserve power and reduce battery drainon the device, thereby extending the device's battery life. Users ofmobile devices may also benefit from arriving at desired content moreefficiently in that the amount of data expended on unsatisfactory videocontent from a limited data plan associated with a user's account on amobile carrier network may be reduced, for example.

In FIG. 1A, a user 102 is shown using a mobile computing device 104 tosubmit video search queries to a video server 106. The mobile computingdevice 104 may be a smartphone, a tablet computing device, a netbook, anotebook computer, or a wearable device, for example. Although notshown, the user may alternatively submit queries from other types ofcomputers as well such as a desktop computer, a television device thatis enabled to stream online video content, or the like. In the exampleof FIG. 1A, the user 102 submits the search queries to a video server106 of a search system that provides a video-sharing service, but thatis not enabled to use watch-time heuristics or creator performanceheuristics (as determined based at least in part on watch-time data) inselecting video content to serve to the computing device 104 in responseto the search requests. The video servers 106 may generally comprise acollection of one or more computers and may perform processing acrossthe one or more computers in one or more geographically separatedlocations.

From the time the user 102 initiates a video search session in theexample of FIG. 1A, the user 102 submits queries for three searchrequests before eventually obtaining satisfactory video content. Therespective search requests and responses are represented as stages110-120 in FIG. 1A. Thus, at stage 110, the user 102 submits via his orher mobile device 104 a query for a first search request 110 to thevideo server 106. At stage 112, the video server 106 provides contentfor one or more videos to the user 102 responsive to the first searchrequest. The user 102 may review the initial video content provided bythe server 106 and determine that he or she is generally not interestedin those videos (i.e., the videos are unsatisfactory). In someimplementations, the video server 106 may return, as a response to asearch request, lists of multiple search results that respectivelyidentify different ones of a plurality of videos that the video server106 has determined to be relevant to the request. The user 102 mayselect search results from the list to cause the mobile device 104 tobegin streaming and playing a particular video that is referenced by theselected search result.

In some instances, the user 102 may “surf” through search results untilfinding satisfactory content by selecting one search result afteranother and watching only short clips of the corresponding videos untilthe user 102 has seen enough to decide whether he or she wishes to watchan extended amount of the video (e.g., because the video is satisfactoryand pertains to subject matter the user was interested in viewing) orwishes to find another video. In some cases, if the user 102 does notfind a satisfactory video in response to a first query, the user mayrefine the query and submit an updated search. The process of surfingthrough a new list of video results may be repeated, and an updatedsearch is performed if no satisfactory video is identified. Thus, atstage 114, the user 102 submits a second video query, and receives asecond response at stage 116. Not until the user 102 submits a searchrequest with a third query, at stage 118, does the video server 106respond with results that identify one or more videos that the user 102finds satisfactory (as indicated by the checkmark by stage 120). Eachnew search request submitted and selected search result that leads towatching a video consumes CPU cycles at the video server 106, consumesnetwork bandwidth, and contributes to battery drain on the mobile device104 because the user watches portions of videos that are not what theuser 102 ultimately intended to view.

Unlike the scenario in FIG. 1A, the scenario represented in FIG. 1Bprovides a video server 156 of a computing system that is enabled to useat least one of a watch-time model 158 and a creator model 160 asheuristics to determine video content to present to a specific user 152(e.g., in response to a search request or as a recommendation).Generally, in this scenario, the user 152 uses his or her mobile device154 to submit a video query to the video server 156 (stage 122), and thevideo server 156 provides one or more video results as a response to thesearch request (stage 124). In this instance, the initial video resultsthat the server 156 sends to the mobile device 154 are improved relativeto the initial results that the server 106 provided in the scenario ofFIG. 1A. For example, the user 152 may even find satisfactory videocontent, or a particular video that he or she desired to locate, in theserver's 156 response to the first query of the video search session.Generally, the server 106 is able to identify more relevant videos forthe user 152 as a result of using the watch-time and/or creators'performance heuristics. Therefore, the user 152 need not surf through asmany video clips or submit as many refined queries to locatesatisfactory content. Accordingly, the user's 152 video search sessionin FIG. 1B places less computational strain on the video server 156,uses less network resources/bandwidth, and reduces unnecessary CPUcycles and battery drain at the mobile device 154.

Notably, the scenarios depicted in FIGS. 1A and 1B represent advantagesthat may be achieved generally through the use of watch-time andcreators' performance heuristics at a video server system. Users may notalways be served satisfactory content in response to an initial search,however, but in the aggregate experience across many users and overtime, these heuristics can in some implementations described hereinfacilitate better, earlier identification of relevant video content toserve to users. This in turn can result in numerous advantages as havebeen described.

FIG. 2 depicts a block diagram of an example environment 200 in whichvideo watch-time data can be captured, analyzed, and shared betweencomputer systems. A data communication network 202 enables datacommunication between multiple electronic devices and systems. In thisenvironment 200, users can access video content, provide video content,exchange information, and search for videos over the network 202. Thenetwork 202 can include, for example, a local area network (LAN), acellular phone and data network, a wide area network (WAN), e.g., theInternet, or a combination of them. The links on the network can bewireline or wireless links or both. Video content creators can alsogenerate and create videos to share over the network 200 using videocreator systems 226.

In some implementations, the environment 200 includes a video searchsystem 210. In operation, the video search system 210 can implement avideo-sharing service that enables various parties (creators) to makedigitized videos available for online distribution to other parties(viewers). The video-sharing service may be accessed from a website orapplications associated with one or more domains. For example, partiesmay upload original video content to the video search system 210, andthe video search system 210 can store and index the videos in the videostorage system 211. Viewers at viewer systems 218 may then query thevideo search system 210 to request video content according to one ormore preferences of the viewers. In some implementations, the videosearch system 210 is configured to provide video content to users basedon indications of how long various videos have been viewed over a periodof time by groups of similarly situated users. The details of the videosearch system 210 and video storage system 211 are described furtherherein with respect to FIG. 3.

Generally, each viewer can access videos through a viewer system 218associated with the respective viewer. A given viewer system 218 caninclude an electronic device, or collection of devices, that are capableof requesting, receiving, and playing videos over the network 202.Example viewer systems 218 may include one or more of a smartphone, atablet computing device, a notebook computer, a desktop computers, asmart television device, a wearable computing device, a virtual realitydevice, an augmented reality device, or a combination of two or more ofthese. The viewer system 218 may include a user application, e.g., a webbrowser or a native media player application that sends and receivesdata over the network 202, generally in response to user actions. Theweb browser can enable a user to display and interact with text, images,videos, music and other information typically located on a web page at awebsite on the Internet or a local area network. The media playerapplication may play digitized videos downloaded or streamed from thevideo search system 210, and may generate watch-time reports that aretransmitted back to the video search system 210 to identify how viewerswatched served videos on the viewer systems 218.

In some implementations, the environment 200 can include a publisherwebsite 204 includes one or more resources 205 associated with a domainand hosted by one or more servers in one or more locations. Generally, awebsite is a collection of web pages formatted in hypertext markuplanguage (HTML) that can contain text, images, multimedia content (e.g.,videos), and programming elements, for example, scripts. Each website204 is maintained by a content publisher, which is an entity thatcontrols, manages and/or owns the website 204. The publisher websites204 can provide a variety of different web pages, such as web pages thatpresent videos hosted by the video-sharing service at video searchsystem 210.

A resource 205 in this context can include any data that is provided bya publisher website 204 over the network 202 and that has a resourceaddress, e.g., a uniform resource locator (URL). Resources may be HTMLpages, electronic documents, images files, video files, audio files, andfeed sources, to name just a few. The resources may include embeddedinformation, e.g., meta information and hyperlinks, and/or embeddedinstructions, e.g., client-side scripts.

In some implementations, the environment 200 can include a content itemmanagement system 220, which generally provides supplemental contentitems (e.g., advertisements) for presentation with videos that the videosearch system 210 serves to viewer systems 218. In some implementations,the content item management system 220 allows supplemental contentproviders to define selection rules that take into accountcharacteristics of a particular video viewer to provide relevantsupplemental content to the viewer. Example selection rules includekeyword selection, in which the supplemental content providers providebids for keywords that are present in either search queries, videos, orvideo content metadata. Supplemental content items that are associatedwith keywords having bids that result in a supplemental item slot beingawarded in response to an auction can be selected for display insupplemental content slots associated with video content. For example,supplemental content may be presented in a banner format near a videoplaying on a viewer's device or over a portion of the video playing onthe viewer's device. In some implementations, the system may causesupplemental content in the form of a video clip to be presented to aviewer as a lead-in before a requested video is played for a viewer.When a viewer selects a supplemental content item, the viewer's devicecan generate a request for a landing page associated with the selectedcontent.

The supplemental content item management system 220 can include a datastorage system that stores campaign data 222 and performance data 224.The campaign data 222 can store, for example, supplemental contentitems, selection information, and budgeting information for supplementalcontent providers. The performance data 224 can store data indicatingthe performance of the supplemental content items that are served. Suchperformance data can include, for example, click-through rates forsupplemental content items, the number of impressions for supplementalcontent items, and the number of conversions for supplemental contentitems.

In some implementations, the campaign data 222 and the performance data224 can be used as input to a supplemental content item selectionprocedure. In particular, the content item management system 220, inresponse to each request for supplemental content, conducts a selectionprocedure to select items that are provided in response to the request.The system 220 can rank supplemental content items according to a scorethat, in some implementations, is proportional to a value based on asupplemental content item bid and one or more parameters specified inthe performance data 224. The highest ranked supplemental content itemsresulting from the auction can be selected and provided to therequesting user device.

Turning to FIG. 3, a block diagram is shown of an example computingsystem 300, including a video search system 302 and a video storagesystem 304, which uses video watch-time information to determine videocontent to serve to computing devices of various viewers incommunication with the computing system 300. In some implementations,the video search system 210 of FIG. 2 may be implemented as the videosearch system 302 of FIG. 3. In some implementations, the video storagesystem 211 may be implemented as the video storage system 304 of FIG. 3.In some implementations, the computing system 300 may further beconfigured to carry out the processes 400, 500, and/or 600, which arerepresented by the flowcharts in FIGS. 4, 5, and 6, respectively. Eachof the video search system 302 and the video search system 304 may beimplemented on one or more computers in one or more locations. Thecomputers of the systems 302 and 304 may be implemented by a combinationof software and hardware as depicted and described with respect to FIG.7.

In some implementations, the video search system 302 provides an onlinevideo-sharing service in which various parties can upload digitizedvideos to the service to make the videos available for distribution toone or more other parties. For the purpose of this document, the partiesthat submit (e.g., upload) videos for distribution through the serviceare referred to as creators, and the parties that watch videos throughthe service are referred to as viewers. In many cases, “creators” mayinclude parties that organically created their own videos to share withothers, but “creators” may also refer to parties who upload content thatwas actually created by one or more other parties but which the firstparty wishes to share on the service. The video search system 302 mayinclude a creator platform 328 that provides an interface for creatorsto submit and monitor the performance of their videos on the sharingservice. Creators may register with accounts 332 with the service andmay use various tools 330 to facilitate video content creation anddistribution.

In some implementations, the video search system 302 may enforcepolicies and technological restrictions to prevent parties fromdistributing content through the video-sharing service without properauthorization from the original creator of the content. The video searchsystem 302 is generally operable to select video content to provide asrecommendations to users or as responses to search queries submitted byusers, based at least in part on watch-time information that indicateshow long users in different viewer categories have viewed differentvideos. In some implementations, viewers can stream digitized videoshosted by the computing system 300. In some implementations, viewers candownload all or portions of digitized videos hosted by the computingsystem 300 to allow the viewers to watch the videos offline at latertimes, for example.

The video storage system 304 is generally responsible for storing,maintaining, and indexing video content for videos that have been madeavailable for distribution on the video-sharing service. The videostorage system 304 can include a video content repository 34 and anindex 336. The video content repository 304 includes one or moreprocessors and one or more storage devices in one or more locations thatstore video content for a large number of digitized videos. For example,when a creator uploads a video to the video search system 302 forsharing, the video file can be provided to the video storage system 304,processed (e.g., compressed and made to conform to one or more standardresolutions), stored, and indexed for searching. Generally, videocontent may include the actual digitized video itself as well aspertinent metadata about the digitized video. For example, the videocontent repository 334 may identify a title, a short textualdescription, and a creator ID for a given video, and correlate themetadata with the digitized video file in the video storage system 304.The index 336 includes information that makes the video contentsearchable, such as references to the identified metadata for variousvideos, hash tables, or the like. The video storage system 304 and thevideo search system 302 can pass messages between each other to identifyand provide video content that is to be served to computing devicesseparate from the system 300 (e.g., over the Internet).

In some implementations, the video search system 300 can include awatch-time modeling apparatus 306, a video content selector 316, aviewer profile manager 318, a network interface (front-end server) 324,a request manager 326, a creator platform 328, or a combination of allor some of these components. Each of the components may generally beimplemented as a combination of hardware and software of one or morecomputers in one or more locations, such as computers described withrespect to FIG. 7.

The network interface 324 is generally configured to enable networkcommunications for the video search system 302. The network interface324 can receive from creators' computing devices requests to makedigitized video content available for distribution on the sharingservice provided by the search system 302. The network interface 324 canalso receive from viewers' computing devices requests to provide sharedvideo content for presentations to the viewers and can serve videocontent to the viewers' computing devices responsive to their requests.

The request manager 326 is generally configured to process requestsreceived from computing devices remote from the video search system 302,as indicated by the network interface 324. For requests for videocontent from viewers' computing devices, the request manager 326 cananalyze the request to identify one or more selection criteria for videocontent that is to be served to the viewers' computing devicesresponsive to the requests. The selection criteria may be expresslyindicated in the content of a given request and/or the selectioncriteria may be identified from data sources external to the request,based on information associated with the request. As an example, somerequests may expressly include a search query that identifies one ormore terms entered and submitted by a viewer, where the terms indicatetopics of video content that the user has targeted for a search.

Some requests, however, may not include a search query or otherwise maynot expressly identify topics of video content that has been requested.The request manager 326 may nevertheless identify topics or otherselection criteria for the request based on circumstantial data ormetadata associated with the request, such as an identity of the user towhom the video content is to be presented, a timestamp assigned to therequest indicating a time that the request was submitted, or locationinformation that indicates a location of the user to whom the requestedvideo content is to be submitted. For example, the request manager 326can identify the targeted viewer based on analysis of the request, andthen one or more characteristics of the user can be identified from theviewer profile manager 218 and used by the video content selector 316 asselection criteria for determining video content to serve in a responseto the request. A video content request may not include a query, forexample, when the video search system 302 is requested to provide arecommendation for video content to a user, such as when the user firstaccesses a homepage of the video-sharing service and before the user hasentered a query, so that the user is automatically presented withoptions for viewing digitized videos of potential interest to the usermerely by virtue of having visited the homepage.

The watch-time modeling apparatus 306 is generally configured todetermine models for scoring videos, creators, or both based onwatch-time data that identifies how long various viewers havingdifferent characteristics have viewed various digitized videos hosted bythe video search system 302 (and stored on the video storage system304). As with other components and sub-components of the video searchsystem 302, the modeling apparatus 306 can comprise one or morecomputers in one or more locations, including one or more processorsand/or one or more computer-readable storage devices. The modelingapparatus 306 can include a watch-time data repository 308, aninstrumentation engine 310, one or more watch-time models 312, one ormore creator performance models 314, or a combination of these.

The watch-time repository 308 stores watch-time data, received at thenetwork interface 324, which identifies how long various viewers haveviewed videos hosted and presented by the video search system 302. Insome implementations, the watch-time repository 308 can include adatabase that logs watch-time reports provided to the video searchsystem 302 from respective viewers' computing devices. Each time awatch-time report is received from a viewer's computing device, an entrycan be added or updated in the database of the watch-time datarepository 308. In some implementations, video playback applications onviewer's computing devices can be configured to automatically generateand provide watch-time reports to the video search system 302 as theviewer watches a given video and/or after the viewer has completedwatching all or a portion of a given video.

For example, when a viewer begins playing a video, the video playbackapplication can automatically transmit a timestamp indicating a starttime of the video to the video search system 302, which is logged in thedatabase. As the viewer continues to watch the video, the video playbackapplication can periodically (e.g., every 1 second or less frequently)ping the video search system 302 to confirm that the user is continuingto play the video. When the viewer stops playing the video, the videoplayback application can send a message to the video search system 302that indicates the viewer has stopped the video, and the message can belogged in the database to indicate the total watch time of the video bythe viewer in a session. In some implementations, the video playbackapplication may store logs of one or more videos a user viewed andrespective watch times of the videos over a period of time (e.g., over abrowsing session, an hour, a day, a week, a month, a year, etc.). Thestored logs can then be transmitted to the video search system 302 on aregular basis, and the modeling apparatus 306 can register the loggeddata in the watch-time data repository 308. In some implementations, thewatch-time data repository 308 may thus include data that identifies aplurality of different digitized videos, and for each respective video,a respective watch time of the video by each of a plurality of viewers.

In some implementations, the modeling apparatus 306 can include aninstrumentation engine 310 (or the instrumentation engine may be aseparate component of the video search system 302). The instrumentationengine 310 is generally configured to generate and inject executablecode or other instructions into web pages or applications associatedwith the playback of videos on client devices to cause the clientdevices to report watch-time data back to the video search system 302.For example, the instrumentation engine 310 may insert a script into aweb page that presents a video, and when the script is executed in a webbrowser at the client computing device, the script monitors the statusof the video being played on the client device and logs watch-time data.The script can then asynchronously report watch-time information to thevideo search system 302.

The modeling apparatus 306 is further operable to generate one or morewatch-time models 312 using information from the watch-time datarepository 308 and information about the viewers whose watch-time isreflected in the watch-time data repository 308. In someimplementations, the modeling apparatus 306 can access or otherwiseobtain information about viewers from the viewer profile manager 318.The viewer profile manager 318 is generally operable to assign uniqueidentifiers to viewers and to correlate information about one or morecharacteristics of viewers with their respective identifiers. In someimplementations, the viewer characteristics can generally be classifiedinto one of two categories, namely demographic characteristics andbehavioral characteristics. The demographic characteristics mayencompass personal characteristics of the viewer (e.g., age group,gender) and/or external characteristics of the viewer (e.g., geographiclocation where video was watched, time of day video was watched, type ofcomputing device on which the viewer watched the video, video playbackapplication used to watch the video, browser application used to watchthe video, viewer's network connection bandwidth).

The behavioral characteristics generally relate to actions the viewertakes in connection with watching a particular video. One example of abehavioral characteristic is a set of one or more queries that the usersubmitted to the video search system and/or to another computing system(e.g., a general search engine) during a user session that led to theviewer watching a particular video. For example, viewer 1 may havevisited the homepage of the video-sharing service provided by the videosearch system 302 and entered a first query of “football.” After seeinga list of video results responsive to the first query, viewer 1 enters arefined query of “hail mary.” The video search system 302 returns, toviewer 1's computing device, as video content responsive to the secondquery, a second list of video search results, from which viewer 1selects a first video to watch. Tom's computing device can generate awatch-time report that is sent to the video search system 302, includinginformation that identifies the first video, information that identifiesviewer 1 (e.g., viewer 1's unique ID), information that identifies howlong viewer 1 viewed the first video, and information that identifiesthe first and second queries. The video search system 302 can thenprocess and store all or some of the information from the report in thewatch-time data repository 308, the behavioral data repository 322, orboth. As such, the system 302 can correlate the watch time for the firstvideo with a viewer category associated with the first query, the secondquery, or both (or keywords extracted from the first and/or secondqueries). Other examples of behavioral characteristics include viewernavigation data, which identifies one or more web pages that the uservisited in a session that led the viewer to a given video; view historydata, which indicates one or more other videos a user viewed in asession in which the user also viewed a particular video that is thesubject of certain watch-time data; click data or conversion data ofcontent items (e.g., ads) that the user selected in connection withviewing a particular video. In some implementations, the viewer profilemanager 318 may store data that identifies one or more content targetingparameters that were used to select video content or other content toserve to a viewer. The viewer profile manager 318, the modelingapparatus 318, or both, may correlate specific watch time segments withbehavioral characteristics of the viewer that generated the watch time.In some implementations, the data managed by the demographic datarepository 320 and the behavioral data repository 322 may be stored inone or more databases on devices in one or more locations.

In some implementations, the viewer profile manager 318 may define aplurality of viewer categories by grouping viewers according to theircharacteristics, as indicated by the demographic data repository 320and/or the behavioral data repository 322. In some implementations, eachunique viewer is assigned to only one of the plurality of viewercategories (i.e., the respective sets of characteristics of the viewercategories is non-overlapping). As a rudimentary example, four viewercategories may be defined based on respective combinations of the agegroup characteristic and the gender characteristic, where eachcharacteristic has two possible alternative values. A first group mayconsist of male viewers aged 25-35, a second group may consist of femaleviewers aged 25-35, a third group may consist of male users over age 35,and a fourth group may consist of female users over age 35. Of course,as the profile manager 318 defines viewer categories based on increasingnumbers of characteristics and/or increasing numbers of possible valuesfor those characteristics, the total number of viewer categories mayincrease rapidly. For example, by segmenting users into one of 5 agegroups (rather than 2), the number of viewer categories may increasefrom 4 to 10. Generally, the viewer categories may be defined ascoarsely or granularly as needed in light of the available viewer data.For example, combinations of tens or hundreds of characteristics may beused to define very granular viewer categories, or combinations of justa few characteristics may be used to define coarser viewer categories.

In some implementations, the viewer profile manager 318 may define aplurality of viewer categories by grouping viewers according to theircharacteristics, where at least some of the viewer categories partiallyoverlap each other. As such, the profile manager 318 may assign a singleviewer to two or more partially overlapping viewer categories asappropriate. For example, a first viewer category may consist of viewersthat (1) arrived at a video by submitting a search query having a firstkeyword and (2) are aged 21-25 years old. A second viewer category mayconsist of viewers that (1) arrived at a video by submitting a searchquery having the first keyword and (2) are female. Therefore, viewer 2,a female viewer between 21-25 years old who, in this example, arrived ata video by submitting a search query having the first keyword, fits intoboth the first and second viewer categories because they are notmutually exclusive categories.

Viewer characteristics and other viewer profile information may beobtained by any of various techniques or combinations of techniques. Insome implementations, viewers may maintain accounts with the videosearch system 302, and users may voluntarily provide demographicinformation to the video search system 302 as part of their accountdata. In some implementations, viewer information may be derived fromcommunications received from viewers' computing devices, includingrequests for video content and watch-time reports sent from viewers'computing devices. For example, location data may be included or derivedfrom messages received from viewer's computing devices, and based on thelocation data a geographic location can be correlated with a viewer. Insome implementations, the video search system 302 may obtain viewerinformation from external sources other than the viewers' computingdevices themselves. For instance, the video search system 302 may obtainsocial data from social networks or may otherwise determine informationabout viewers from web pages or other publicly available documents onthe Internet.

In situations in which the systems and other techniques discussed herecollect personal information about users (e.g., viewers), or may makeuse of personal information, the users may be provided with anopportunity to control whether programs or features collect userinformation (e.g., information about a user's watch time, socialnetwork, social actions or activities, profession, a user's preferences,a user's search history, a user's navigation history, or a user'scurrent location), or to control whether and/or how to receive contentfrom the video server that may be more relevant to the user. Inaddition, certain data may be treated in one or more ways before it isstored or used, so that personally identifiable information is removed.For example, a user's identity may be treated so that no personallyidentifiable information can be determined for the user, or a user'sgeographic location may be generalized where location information isobtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby the system.

Referring again to the modeling apparatus 306 of the video search system302, the modeling apparatus 306 may generate, store, and maintain (e.g.,update) one or more watch-time models 312 based on video watch-timedata, as indicated by the watch-time data repository 308, and based oncharacteristics of viewers whose activities produced the watch-timedata, as indicated by the viewer profile manager 318. Generally, thewatch-time models 312 store data that indicate how long various groupsof viewers watched individual videos or groups of videos. In someimplementations, the modeling apparatus 306 generates the watch-timemodels 312 by grouping the watch time for respective videos or groups ofvideos based on characteristics of the viewers of the respective videosor groups of videos. In some implementations, the modeling apparatus 306identifies appropriate viewer groups from the viewer profile manager318. Therefore, the viewer groups employed by the modeling apparatus 306may correspond to the viewer categories defined by the viewer profilemanager 318, such that the watch-time models 312 indicate how longvarious viewers within each of the viewer categories watched individualvideos or groups of videos. For example, the watch-time data repository308 may show that, over a certain period of time (e.g., an hour, a day,a week, or a month) 5,000 unique viewers watched a particular video foran accumulated watch time of 8 hours among all the viewers. Thewatch-time models 312 may, in turn, specify a distribution of the watchtime for the particular video among various viewer categories, asindicated by the viewer profile manager 318. For instance, thewatch-time model 312 may indicate that 37 minutes of the 8 hours oftotal watch time are associated with male viewers located in urbangeographic areas, 4.5 hours of the total watch time are associated withfemale viewers in rural geographic areas, 70 minutes of the total watchtime are associated with male viewers in rural geographic areas, and thebalance of the 8 hours of watch time is associated with associated withfemale viewers located in urban geographic areas. The watch-time model312 thus indicates for the particular video relative interests in thevideo by different categories of viewers as indicated by the relativewatch times of the video by viewers in each category. Similarly, themodeling apparatus 306 may boost the watch time of viewers who haveachieved a certain status on the video-sharing service or a relatedservice (e.g., a social network) as a reward to those viewers forachieving the status or because the status signifies a level of trust inthe viewers' viewing habits. For example, the watch times of registeredmembers of the video-sharing service may be boosted relative to thewatch times of non-registered viewers.

As previously described, the viewer profile manager 318 can in someimplementations define partially overlapping viewer categories such thata given viewer can belong to multiple different viewer categories. Insuch implementations, the modeling apparatus 306 may apportion theviewer's watch time according to among each of the viewer categories towhich the viewer belongs according to various criteria. For example, themodeling apparatus 306 may assign 40-percent of a viewer's watch time toa first viewer category to which the user belongs, and can assign theremaining 60-percent of the viewer's watch time to a second viewercategory to which the user belongs. In some implementations, theapportionment of a viewer's watch time among each applicable viewercategory can be based on scores assigned to the applicable viewercategory. The scores may reflect, for example, the relative values ofeach viewer category to the video search system 302. For example, agiven viewer may belong to both first and second viewer categories, towhich the modeling apparatus 306 has assigned scores of 5 and 10,respectively. Therefore, according to the viewer category scores, ⅓ ofthe viewer's watch time may be apportioned to the first viewer categoryand ⅔ of the viewer's watch time may be apportioned to the second viewercategory. In some implementations, the apportionment of a viewer's watchtime among each applicable viewer category can be based on otherinformation about the viewer or about the data that was used to classifya user into one or more viewer categories. For example, the viewerprofile manager 218 may process data about a viewer and may determinethat there is a 75-percent likelihood that the viewer is male and a25-percent likelihood that the viewer is female. Because the viewerprofile manager 218 does not have complete confidence in the viewer'sgender classification, 75-percent of the viewer's watch time may beassigned to a viewer category defined at least in part by a malecharacteristic, and 25-percent of the viewer's watch time may beassigned to a viewer category defined at least in part by a femalecharacteristic.

In some instances, it may be useful to keep track of viewer's watchtimes with respect to groups of videos rather than or in addition toindividual videos. As such, the modeling apparatus 306 may, in someimplementations, determine watch-time models 312 that indicate, for eachof multiple groups of videos, a distribution of watch times of the videoamong various viewer categories. For example, the modeling apparatus 306may analyze data from the watch-time data repository 308 to determinetotal watch times of videos within respective groups of videos over aperiod of time by a population of viewers. The modeling apparatus canthen group the watch time for each group of video by viewer category togenerate watch-time distributions. The modeling apparatus 306 may groupvideos according to various criteria, as indicated by the video storagesystem 304. For example, videos may be grouped by creator, by channel,by age (e.g., amount of time since a video was submitted fordistribution on the video-sharing service), by genre (e.g., productreviews, music videos, animation, television shows, action, comedy,horror, children's videos), by popularity (e.g., total number of views),or a combination of two or more of these. By determining watch-timedistributions for groups of videos, the video selector 316 can, in someimplementations, more readily determine video content to serve to aviewer's computing device in response to a request for video content byselecting videos from within one or more groups of videos that haverelatively high watch times by viewers that have characteristics thatmatch characteristics of the viewer to whom the selected video contentis to be presented.

As the video search system 302 may constantly collect new watch-timedata from viewers, the modeling apparatus 312 may be configured toupdate or regenerate the watch-time models 312 (and the contentperformance models 314) on a continuous or periodic basis. In someimplementations, the models 312, 314 may be maintained based on arolling window of watch-time data. For example, once each day themodeling apparatus 312 may update the models 312, 314 based on watchtime that occurred in the past 7 days. Every day, then, the models 312,314 can be updated to incorporate watch-time data from a most recent dayand to discard watch-time data more than a week old. In someimplementations, the modeling apparatus 306 can update the models 312,314 with a completely fresh set of data (e.g., every week the models maybe regenerated using data from only the most recent week). In someimplementations, the modeling apparatus 312, 314 may update the modelsfrom time to time to incorporate a most recently collected set ofwatch-time data without discarding older watch-time data.

The watch-time models 312 may organize the groupings of video watch-timeinformation in various ways. In some implementations, each respectivevideo or group of videos can be correlated with a plurality of valuesthat respectively indicate the total (accumulated) watch time of therespective video or group of videos by viewers in a respective one of aplurality of viewer categories. In some implementations, the watch-timemodels 312 may indicate the converse. Namely, for each of a plurality ofviewer categories, watch times of viewers within the respective viewercategory may be distributed among a set of videos or groups of videos.

In some implementations, the watch-time models 312 can indicate, foreach video or group of videos served by the video search system 302 overa period of time, a distribution of watch times for the respective videoor group of videos among each of a plurality of viewer categories. If noviewers have viewed a given video within a particular category duringthat period of time, the watch time assigned to that category may benull (zero). In some implementations, the watch-time models 312 mayidentify actual watch time totals for each viewer category (e.g.,viewers in a first category watched the video for a total of 132minutes, while viewers in a second category watched the video for atotal of 61 minutes). In some implementations, the watch-time models 312may identify relative watch time totals for each viewer category (e.g.,68-percent of the watch time for the video was by viewers in the firstcategory, while 32-percent of the watch time for the video was byviewers in the second category). In some implementations, the videosearch system 302 may value certain viewers' watch time more thanothers, and therefore the modeling apparatus 306 may weigh the watchtime of individual viewers or groups of viewers when determining thewatch-time distributions for the watch-time models 312. For example, acelebrity or an expert in a field that is the subject of a video orgroup of videos may have their actual watch times tripled for the videoor group of videos, or other users' watch times may be devalued.

In some implementations, the modeling apparatus 306 can use videowatch-time data to determine one or more creator performance models 314.Generally, the creator performance models 314 identify creatorperformance scores (i.e., scores for parties that have submitted videosto the video-sharing service for distribution). The creator performancescores can be determined by the modeling apparatus 306 based on how longvarious categories of viewers have viewed the creators' videos. In thisway, the video search system 302 can leverage watch-time information asa metric for assessing the performance of creators on the video-sharingservice. Moreover, and as further described with respect to the videocontent selector 316, the creator performance scores can be used in someimplementations as a heuristic for ranking and determining video contentto serve to viewers. In some implementations, the video search system302 may also determine how to allocate assets to creators based at leastin part on the performance scores.

In some implementations, the modeling apparatus 306 may determinecreator performance scores as follows. First, the modeling apparatus 306accesses from the watch-time data repository 308 information about howlong viewers have watched various videos over a period of time. Themodeling apparatus 306 then identifies from the viewer profile manager318 a set of one or more viewer categories that will form the basis ofthe creator performance scores. In some implementations, the identifiedset of viewer categories may be a complete set of viewer categories thatencompasses all viewers, or the identified set of viewer categories maycomprise less than all of the viewer categories in the complete set. Forexample, if the complete set of viewer categories included (1) malesover 45, (2) males 35-45, (3) females over 45, and (4) females 35-45,then the modeling apparatus 306 could determine the creator performancescores based on the watch times of viewers within all four categories(the complete set), or based on the watch times of viewers within fewerthan all four categories.

The modeling apparatus 306 then generates groups of watch time for eachof the identified viewer categories by assigning respective pieces ofwatch time to appropriate ones of the viewer categories based on thecharacteristics of the viewers whose views resulted in the respectivewatch times. For example, the watch times of one or more videos by oneor more viewers who belong to a first viewer category may be assigned toa watch time group for the first viewer category, the watch times of oneor more videos by one or more viewers who belong to a second viewercategory may be assigned to a watch time group for the second viewercategory, and so on. Based on the groupings, the modeling apparatus 306then accumulates the watch times in each viewer category to determine atotal watch time that indicates, for each viewer category, a totalamount of time that viewers within the category spent watching videosover a defined time interval (e.g., a day, a week, a month). Themodeling apparatus 306 also breaks down the total watch time in eachviewer category by creator. That is, in each viewer category, themodeling apparatus 306 identifies all the creators of the videos watchedby viewers within the category, and determines for each of theidentified creators how much of the total watch time for the categorywas watch time of videos of the respective creator. As an example, themodeling apparatus 306 may determine that, over the course of a month,viewers aged 45-55 watched a total of 2,000 hours of video, whileviewers aged 56-65 watched a total of 4,000 hours of video over thatmonth. Moreover, the total watch time of multiple videos distributed bya first creator on the video-sharing service over that month by viewersaged 45-55 may be determined as 130 hours. The total watch time of themultiple videos distributed by the first creator on the video-sharingservice over that month by viewers aged 56-65 may be determined as 25hours. Thus, the first creator's videos can be seen to contribute to agreater share of the total watch time for the 45-55 category than the56-65 category.

Further in the process of determining creator performance scores, themodeling apparatus 306 can identify scores for each of the viewercategories that indicate how much weight the modeling apparatus 306 willafford watch time from each of the viewer categories in determiningcreator performance scores. Watch time from different viewer categoriesmay be weighted differently from each other, for example, to rewardcreators whose videos generate more watch time from viewer categoriesthat the video-sharing service deems more valuable than from viewercategories that the video-sharing service deems less valuable. Thus, ifthe video-sharing service targets video content to particulardemographics of viewers, creators can receive more credit for watch timegenerated by viewers within the targeted demographics than for watchtime by viewers in other demographics. In some implementations, themodeling apparatus 306 may determine the viewer category scores based onhow much revenue viewers within the respective viewer categoriesgenerated for the video-sharing service over a period of time by playingvideos distributed through the video-sharing service during that periodof time. For example, the video search system 302 may serve videocontent for presentation to viewers along with additional sponsoredcontent that third parties pay the video-sharing service to present. Thevideo-sharing service can therefore generate revenue from servingsponsored content, and revenue can be attributed to individual instancesof video content served to viewers and to collections of video contentserved to respective categories of users. In some implementations, themodeling apparatus 306 can assign scores to the various viewercategories that correspond to the revenue generated from servingsponsored content to viewers in the various viewer categories.

Using the viewer category scores and the watch times associated with thevarious viewer categories, the modeling apparatus 306 can compute thecreator performance scores. In some implementations, the creatorperformance score for a given creator can be computed by (1)determining, for each respective viewer category, the product of the (i)share of the total watch time for the respective viewer category that isattributable to videos associated with the given creator and (ii) theviewer category score for the respective viewer category, and (2) takingthe sum of the products across all the viewer categories. For example,consider a scenario in which viewers in a first category watched 100minutes of videos over a period of time and viewers in a second categorywatched 200 minutes of videos over the same period of time. The share ofthe watch time attributable to videos of a particular creator by viewersin the first category is 20 minutes and the share of the watch timeattributable to videos of the same creator by viewers in the secondcategory is also 20 minutes. The viewer category score for the firstvideo is 50 and the viewer category score for the second video is 250.The creator performance score for the particular creator can becalculated as (20/100)*(50)+(20/200)*(250)=35. In some implementations,the video-sharing service can use the creator performance scores as amodel or heuristic for distributing assets (e.g., points, incentives,membership status, access to creative tools, or revenue) to thecreators.

The video content selector 316 is operable to select video content toserve to various computing devices in response to requests for videocontent. Generally, the video content selector 316 can select videocontent to serve based on watch-time information, as indicated by thewatch-time models 312, based on creator performance scores, as indicatedby the creator performance models 314, or both. In response to a requestfrom a particular viewer, the video content selector 316 may selectvideo content for one or more videos to serve to the particular viewer'scomputing device based on identifying that historically the one or morevideos, or other videos that are similar to the one or more videos, werewatched for relatively long times by various viewers havingcharacteristics that match or are similar to characteristics of theviewer for whom the video content is targeted. For example, the videocontent selector 316 may identify, from the request manager 326, that auser who submitted a request for video content is a middle-aged malefrom Albuquerque, New Mexico. The video content selector 316 can thenquery the watch-time models 312 to identify videos that the watch-timedata indicates were preferred (e.g., watched for longer times) by otherusers that match the same profile of the requesting user. Generally,videos having higher watch times by matching or similar viewers are morelikely to be selected in response to a request than are videos havinglower watch times by matching or similar viewers. Continuing thepreceding example, the video content selector 316 may rank a pluralityof videos that are determined to be relevant to the request from themiddle-aged male from Albuquerque. The videos may be ranked based onmultiple heuristics, including how closely the subject matter ofcandidate videos matches one or more topics of the request, watch-timeheuristics, and/or creator performance heuristics. With respect to thewatch-time heuristics, candidate videos can be promoted in the rankingthe longer that the videos were watched by middle-aged men fromAlbuquerque or by viewers in similar demographics. With respect to thecreator performance heuristics, candidate videos by creators havinghigher performance scores may be promoted in the ranking. The videoselector 316 can then select video content for one or more of thetop-ranked candidate videos to serve to the requesting user's computingdevice. In some implementations, the served content may be the digitalvideos themselves. In some implementations, the served content may notinclude the digital videos themselves, but may include references to thedigital videos (e.g., search results that include a title, description,and/or representative image of the selected videos).

In some implementations, the video selector 316 may apply back-offtechniques to identify viewer characteristics (and hence viewercategories) that are similar to characteristics of a user that hasrequested video content. For example, rather than limiting the analysisof watch time by middle-aged male viewers in Albuquerque, New Mexico,the video selector 316 may query the watch-time models 312 to identifywatch times of videos by middle-aged male viewers in the entiresouthwest United States. By expanding the relevant geographic area(e.g., by backing-off), more data points can be analyzed to determinemore reliable results.

Turning to FIG. 4, a flowchart is shown of an example process 400 forgenerating a watch-time model that a video search system can use toaffect the selection of video content to serve to a user based onhistorical information about how long various groups of viewers havewatched different videos. The watch-time model that results from theprocess 400 is generally equivalent to the watch-time models 312 thathas been described with respect to FIG. 3. In some implementations, theprocess 400 may be carried out by the systems and devices discussedthroughout this document, including by the watch-time modeling apparatus306 of FIG. 3.

The process can begin at stage 402, when a computing system obtains aset of watch-time data. The watch-time data can indicate, for each ofvarious videos, how long various viewers have watched the video at theirrespective computing devices. For example, data representing that aviewer has watched a ten-minute video for only five seconds may stronglysuggests that the viewer was not interested in that video. A viewer thatwatched substantially the entire video, however, is much more likely tohave been interested in the video. Watch-time thus serves as a proxy forinferring viewers' levels of interest in videos. The watch-time data canbe reported by the individual viewers' computing devices to a centralcomputing system that generates the watch-time model, in some instances.In some implementations, the computing system may generate thewatch-time model using data about the actual lengths of time viewerswatched various videos. However, to the extent the actual watch timesmay unduly favor longer videos over shorter videos, the computing systemcan, in some implementations, normalize the watch-time data so that itrepresents the portions of the overall video length that viewers watched(e.g., viewer 1 watched 30-percent of video 1, viewer 2 watched78-percent of video 1).

At stage 404, the computing system identifies characteristics of theviewers whose video views are represented in the watch-time data. Thecharacteristics may include demographic characteristics of the viewers,behavioral characteristics of the viewers, other characteristics of theviewers, or a combination of these. At stage 406, the computing systemgroups viewers into a collection of different viewer categories. Eachviewer category can include viewers that share a same set ofcharacteristics with each other. For example, all viewers that arrivedat their respective videos using the same or similar search queries maybe grouped together, or all viewers having the same or similarnavigation histories within a same geographic area may be grouped into aviewer category. In some implementations, the system may group viewersat very granular levels such that all viewers in the groups share thesame set of multiple characteristics. The system may then cluster someof the granular groups together by merging groups whose viewers meet athreshold level of similarity (e.g., viewers having similarcharacteristics or groups having similar watch-time distributions). Atstage 408, the system correlates watch-time information with appropriateviewer categories. For example, watch time for videos watched by viewerswithin a first viewer category can be grouped together, watch time forvideos watched by viewers within a second category can be groupedtogether, and so on. At stage 410, the computing system determinesdistributions of watch time for videos by viewers within each of theviewer categories, and at stage 412, information about thesedistributions is stored in a watch-time model.

FIG. 5 is a flowchart of an example process 500 for selecting videocontent to serve to a computing device of a targeted user usinginformation about how long users similar to the targeted user havewatched various videos during a past period of time. In someimplementations, the process 500 may be carried out by the computingsystems and devices discussed throughout this document, including by thevideo search system 302 discussed with respect to FIG. 3.

At stage 502, the video search system receives a request to serve videocontent to a user. The request may have originated from the user as arequest to search for specific video content, or the request may havebeen generated to provide unsolicited video recommendations to the user,for example. At stage 504, the search system identifies selectioncriteria associated with the request. The selection criteria may beexpressly specified in the request (e.g., keywords in a search query),or may be derived from the request based on information contained in therequest (e.g., characteristics of a user identified in the request). Insome implementations, the system can identify selection criteria thatincludes both video topics (stage 506) and user characteristics (stage508). At stage 510, the video system accesses one or more watch-timemodels, such as those described with respect to FIGS. 3 and 4. Thewatch-time models may, for example, indicate distributions of watch timeof various videos by different categories of viewers. At stage 512, thesystem can determine content for a set of candidate videos to serve tothe user in response to the received request. The candidate videos maybe selected based on the selection criteria (e.g., videos that matchtopics specified in a query) and based on the watch-time distributionsindicated by the watch-time models. At stage 514, the search systemranks the candidate videos according to one or more heuristics todetermine an ordered list of videos that are determined to be mostrelevant to the user. In some implementations, the system may rank thevideos based at least in part on how long all or some of the candidatevideos have been watched by users whose characteristics match or aresimilar to the characteristics of the targeted user. For example, videosthat have been watched for longer times (or for which a greaterproportion of the video was watched) by matching or similar users may bepromoted in the ranking as compared to videos with lower watch times. Atstage 516, the system selects video content for one or more top-rankedvideos to serve to the user, and at stage 518 the system serves theselected video content.

FIG. 6 shows a flowchart of an example process 600 for determining andusing creator performance scores to rank video content and to allocateassets distributed to creators for improving the quality of videossubmitted to a video-sharing service. In some implementations, theprocess 600 may be carried out by the systems and devices discussedherein, including by the video search system 302 of FIG. 3.

At stage 602, a computing system accesses watch-time models thatindicate distributions of watch times of various videos by viewers indifferent viewer categories. In some implementations, the data indicatedby the watch-time models may be used to determine the creatorperformance scores. Additionally or alternatively, the creatorperformance scores may be determined in part based on raw watch-timedata before the data is processed for use in the watch-time models.

At stage 604, the computing system uses the watch-time data to determinea creator performance model. The creator performance model indicatesscores for creators that reflect the relative performance of theirvideos with respect to various viewer categories, based at least in parton how long viewers in the various categories have viewed the creators'respective videos. Watch time from viewers in some categories may bemore valuable than watch time from viewers in other categories, and sothe creator performance model can take into account different weightsthat apply to the various viewer categories when computing the creatorperformance model. In some implementations, operations for determiningthe creator performance model are represented by stages 606-612. Atstage 606, the system determines the total watch times of videos byviewers in each respective viewer category. At stage 608, the systemdetermines, for each viewer category, respective shares of the totalwatch time for the viewer category that are attributable to videos ofvarious creators. At stage 610, the system determines scores for thevarious user categories, where the scores can be used to weight thecontribution of watch time from each viewer category to the creators'performance scores. In some implementations, the viewer category scorescan be based on an amount of revenue that was generated by servingvideos to the viewers in the respective viewer categories. At stage 612,the system determines creator performance scores for each of thecreators based on the shares of watch times attributed to the variouscreators in each viewer category, and based on the viewer categoryscores.

Once the creator performance model has been determined, the system usesthe model, and the scores indicated by the model, to rank creators withrespect to each other (stage 614). The system may then distribute assetsto creators according to the creator performance scores and/or theranking (stage 616). The system may also use the creator performancescores and/or the ranking as a heuristic for selecting content forquality videos to serve to computing devices in response to requests(stage 618).

FIG. 7 shows an example of a computing device 700 and a mobile computingdevice that may be used to implement the computer-implemented methodsand other techniques described herein. The computing device 700 isintended to represent various forms of digital computers, such aslaptops, desktops, workstations, personal digital assistants, servers,blade servers, mainframes, and other appropriate computers. The mobilecomputing device is intended to represent various forms of mobiledevices, such as personal digital assistants, cellular telephones,smart-phones, and other similar computing devices. The components shownhere, their connections and relationships, and their functions, aremeant to be exemplary only, and are not meant to limit implementationsof the inventions described and/or claimed in this document.

The computing device 700 includes a processor 702, a memory 704, astorage device 706, a high-speed interface 708 connecting to the memory704 and multiple high-speed expansion ports 710, and a low-speedinterface 712 connecting to a low-speed expansion port 714 and thestorage device 706. Each of the processor 702, the memory 704, thestorage device 706, the high-speed interface 708, the high-speedexpansion ports 710, and the low-speed interface 712, are interconnectedusing various busses, and may be mounted on a common motherboard or inother manners as appropriate. The processor 702 can process instructionsfor execution within the computing device 700, including instructionsstored in the memory 704 or on the storage device 706 to displaygraphical information for a GUI on an external input/output device, suchas a display 716 coupled to the high-speed interface 708. In otherimplementations, multiple processors and/or multiple buses may be used,as appropriate, along with multiple memories and types of memory. Also,multiple computing devices may be connected, with each device providingportions of the necessary operations (e.g., as a server bank, a group ofblade servers, or a multi-processor system).

The memory 704 stores information within the computing device 700. Insome implementations, the memory 704 is a volatile memory unit or units.In some implementations, the memory 704 is a non-volatile memory unit orunits. The memory 704 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 706 is capable of providing mass storage for thecomputing device 700. In some implementations, the storage device 706may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The computer program product can also be tangiblyembodied in a computer- or machine-readable medium, such as the memory704, the storage device 706, or memory on the processor 702.

The high-speed interface 708 manages bandwidth-intensive operations forthe computing device 700, while the low-speed interface 712 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In some implementations, the high-speed interface 708 iscoupled to the memory 704, the display 716 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 710,which may accept various expansion cards (not shown). In theimplementation, the low-speed interface 712 is coupled to the storagedevice 706 and the low-speed expansion port 714. The low-speed expansionport 714, which may include various communication ports (e.g., USB,BLUETOOTH, Ethernet, wireless Ethernet) may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 700 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 720, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 722. It may also be implemented as part of a rack server system724. Alternatively, components from the computing device 700 may becombined with other components in a mobile device (not shown), such as amobile computing device 750. Each of such devices may contain one ormore of the computing device 700 and the mobile computing device 750,and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 750 includes a processor 752, a memory 764,an input/output device such as a display 754, a communication interface766, and a transceiver 768, among other components. The mobile computingdevice 750 may also be provided with a storage device, such as amicro-drive or other device, to provide additional storage. Each of theprocessor 752, the memory 764, the display 754, the communicationinterface 766, and the transceiver 768, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 752 can execute instructions within the mobile computingdevice 750, including instructions stored in the memory 764. Theprocessor 752 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 752may provide, for example, for coordination of the other components ofthe mobile computing device 750, such as control of user interfaces,applications run by the mobile computing device 750, and wirelesscommunication by the mobile computing device 750.

The processor 752 may communicate with a user through a controlinterface 758 and a display interface 756 coupled to the display 754.The display 754 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface756 may comprise appropriate circuitry for driving the display 754 topresent graphical and other information to a user. The control interface758 may receive commands from a user and convert them for submission tothe processor 752. In addition, an external interface 762 may providecommunication with the processor 752, so as to enable near areacommunication of the mobile computing device 750 with other devices. Theexternal interface 762 may provide, for example, for wired communicationin some implementations, or for wireless communication in otherimplementations, and multiple interfaces may also be used.

The memory 764 stores information within the mobile computing device750. The memory 764 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 774 may also beprovided and connected to the mobile computing device 750 through anexpansion interface 772, which may include, for example, a SIMM (SingleIn Line Memory Module) card interface. The expansion memory 774 mayprovide extra storage space for the mobile computing device 750, or mayalso store applications or other information for the mobile computingdevice 750. Specifically, the expansion memory 774 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 774 may be provide as a security module for the mobilecomputing device 750, and may be programmed with instructions thatpermit secure use of the mobile computing device 750. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. The computerprogram product contains instructions that, when executed, perform oneor more methods, such as those described above. The computer programproduct can be a computer- or machine-readable medium, such as thememory 764, the expansion memory 774, or memory on the processor 752. Insome implementations, the computer program product can be received in apropagated signal, for example, over the transceiver 768 or the externalinterface 762.

The mobile computing device 750 may communicate wirelessly through thecommunication interface 766, which may include digital signal processingcircuitry where necessary. The communication interface 766 may providefor communications under various modes or protocols, such as GSM voicecalls (Global System for Mobile communications), SMS (Short MessageService), EMS (Enhanced Messaging Service), or MMS messaging (MultimediaMessaging Service), CDMA (code division multiple access), TDMA (timedivision multiple access), PDC (Personal Digital Cellular), WCDMA(Wideband Code Division Multiple Access), CDMA2000, or GPRS (GeneralPacket Radio Service), among others. Such communication may occur, forexample, through the transceiver 768 using a radio-frequency. Inaddition, short-range communication may occur, such as using aBLUETOOTH, WiFi, or other such transceiver (not shown). In addition, aGPS (Global Positioning System) receiver module 770 may provideadditional navigation- and location-related wireless data to the mobilecomputing device 750, which may be used as appropriate by applicationsrunning on the mobile computing device 750.

The mobile computing device 750 may also communicate audibly using anaudio codec 760, which may receive spoken information from a user andconvert it to usable digital information. The audio codec 760 maylikewise generate audible sound for a user, such as through a speaker,e.g., in a handset of the mobile computing device 750. Such sound mayinclude sound from voice telephone calls, may include recorded sound(e.g., voice messages, music files, etc.) and may also include soundgenerated by applications operating on the mobile computing device 750.

The mobile computing device 750 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 780. It may also be implemented aspart of a smart-phone 782, personal digital assistant, or other similarmobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term machine-readable signal refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

In situations in which the systems, methods, devices, and othertechniques here collect personal information (e.g., context data) aboutusers, or may make use of personal information, the users may beprovided with an opportunity to control whether programs or featurescollect user information (e.g., information about a user's socialnetwork, social actions or activities, profession, a user's preferences,or a user's current location), or to control whether and/or how toreceive content from the content server that may be more relevant to theuser. In addition, certain data may be treated in one or more waysbefore it is stored or used, so that personally identifiable informationis removed. For example, a user's identity may be treated so that nopersonally identifiable information can be determined for the user, or auser's geographic location may be generalized where location informationis obtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby a content server.

Although various implementations have been described in detail above,other modifications are possible. In addition, the logic flows depictedin the figures do not require the particular order shown, or sequentialorder, to achieve desirable results. In addition, other steps may beprovided, or steps may be eliminated, from the described flows, andother components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

What is claimed is:
 1. A computing system for serving videos over anetwork, the computing system comprising: a modeling apparatus that (i)obtains, for various digitized videos that the computing system haspresented at client devices over a network, watch time information thatspecifies an amount of time that the various digitized videos werepresented at the client devices, (ii) groups the watch time informationinto different groups based on characteristics of users associated withthe client devices, and (iii) determines, based on the grouping of watchtime information and for each particular group of users of a pluralityof groups of users, an accumulated watch time that indicates how longthe various digitized videos were presented to users within theparticular group of users; a front-end server that receives a requestfor video content and serves video content identified in response to therequest over the network to a computing device that is separate from thecomputing system for presentation at the computing device; a requestmanager, including one or more processors, that analyzes the request forvideo content and identifies selection criteria in the request,including identifying one or more characteristics of a user thatsubmitted the request for video content; one or more video file storagedevices that store a plurality of digitized videos that have been madeavailable by various parties for distribution over the network; and avideo selector, including one or more processors, that selects, from thevideo file storage devices, video content of a particular digitizedvideo among the plurality of digitized videos to serve to the computingdevice in response to the request, wherein the selection is based atleast in part on a first accumulated watch time that indicates how longthe particular digitized video has been presented at client devicesassociated with users having characteristics that match the one or morecharacteristics of the user that submitted the request for videocontent, as indicated by the modeling apparatus.
 2. The computing systemof claim 1, wherein the modeling apparatus: groups the watch timeinformation into different groups by assigning respective portions ofthe watch time information to different viewer categories thatcorrespond to different sets of characteristics of the users associatedwith the client devices at which the various digitized videos werepresented; and identifies, for each respective viewer category, adistribution of watch time of users in the respective viewer categoryamong digitized videos in a group of digitized videos that were at leastpartially presented at computing devices associated with users in therespective category.
 3. The computing system of claim 1, wherein, foreach respective digitized video among the plurality of digitized videosstored on the one or more video file storage devices, the modelingapparatus: identifies, from the watch time information and for eachrespective viewer category among a plurality of viewer categories, howlong the respective digitized video was presented at client devicesassociated with users in the respective viewer category; and assigns tothe respective digitized video, for each respective viewer categoryamong the plurality of viewer categories and based on how long therespective digitized video was presented at client devices associatedwith users in the respective viewer category, a score that indicates arelevance of the respective digitized video to the respective viewercategory.
 4. The computing system of claim 1, wherein the one or morecharacteristics of the user that submitted the request for video contentcomprise at least one of demographic characteristics of the user orbehavioral characteristics of the user.
 5. The computing system of claim4, wherein the one or more characteristics of the user that submittedthe request for video content comprise demographic characteristics thatindicate at least an age or gender of the user.
 6. The computing systemof claim 4, wherein: the one or more characteristics of the user thatsubmitted the request for video content comprise behavioralcharacteristics that indicate one or more queries submitted by the userin a same session in which the user submitted the request for videocontent; and the video selector selects to serve the video content ofthe particular digitized video among the plurality of digitized videosbased at least on how long the particular digitized video was presentedat client devices associated with other users during sessions in whichthe other users also submitted the one or more queries or at least onequery determined to be similar to the one or more queries.
 7. Thecomputing system of claim 4, wherein: the one or more characteristics ofthe user that submitted the request for video content comprisebehavioral characteristics that indicate a navigation history of theuser associated with the request for video content; and the videoselector selects to serve the video content of the particular digitizedvideo among the plurality of digitized videos based at least on how longthe particular digitized video was presented at client devicesassociated with other users having a same or similar navigation historyas the user that submitted the request, in browsing sessions that ledthe other users to watch the particular digitized video.
 8. Thecomputing system of claim 1, wherein the video selector is furtherconfigured to select video content to serve to various computing devicesas recommendations to respective users of the various computing devices.9. The computing system of claim 1, wherein: in response to receivingthe request for video content, the request manager is configured toidentify one or more topics of the requested video content, includingidentifying the topics from at least one of a user-entered query that isa part of the request or identifying the topics from informationassociated with the user that is not included in the request; and thevideo selector is configured to select video content to serve to thecomputing device by: identifying digitized videos that are determined tobe relevant to the one or more topics; ranking the identified digitizedvideos based at least on how long each of the identified digitizedvideos has been presented at client devices associated with users havingcharacteristics that match the one or more characteristics of the userthat submitted the request for video content; and based on the ranking,selecting video content of one or more top-ranked digitized videos toserve to the computing device, including the video content of theparticular digitized video.
 10. The computing system of claim 1, furthercomprising an instrumentation manager, including one or more processors,that generates and embeds computer code in various web pages that thecomputing system serves to various client devices for the presentationof various digitized videos, wherein the computer code is configured,when executed on the various client devices, to cause the variouscomputing devices to periodically report over the network and to thecomputing system amounts of time that the various digitized videos werepresented at the various client devices.
 11. The computing system ofclaim 1, wherein: the front-end server is configured to serve the videocontent of the particular digitized video by streaming the video contentto the computing device, and the computing device is configured to (i)receive the stream of the video content of the particular digitizedvideo from the computing system and (ii) play the stream of the videocontent.
 12. A computer-implemented method, comprising: obtaining, at acomputing system, data that indicates, for each respective computingdevice among a plurality of computing devices to which at least aportion of a first video hosted by the computing system was served, arespective watch time for the first video that occurred at therespective computing device; for each of the respective watch times forthe first video, attributing the respective watch time to one or moreviewer categories among a plurality of viewer categories based on one ormore characteristics of a user of the respective computing device atwhich the respective watch time occurred; for each respective viewercategory among the plurality of viewer categories, accumulating therespective watch times that have been attributed to the respectiveviewer category to generate an accumulated watch time for the respectiveviewer category, wherein the accumulated watch time for the respectiveviewer category indicates how long the first video was presented tousers within the respective viewer category; determining whether toprovide video content of the first video to a first user using acriterion that is based at least on the accumulated watch time for thefirst video for a particular viewer category among the plurality ofviewer categories that matches the first user; and in response to thecomputing system determining that the criterion for providing the videocontent of the first video to the first user is satisfied, serving, to afirst computing device of the first user, the video content of the firstvideo for presentation at the first computing device.
 13. Thecomputer-implemented method of claim 12, wherein determining whether toprovide video content of the first video to the first user comprisesdetermining whether the accumulated watch time of the first video forthe particular viewer category that matches the first user meets athreshold watch time.
 14. The computer-implemented method of claim 12,wherein determining whether to provide video content of the first videoto the first user comprises: ranking a plurality of candidate videos,including the first video, based at least on respective accumulatedwatch times of the plurality of candidate videos for the particularviewer category that matches the first user; and selecting video contentfor one or more videos of the plurality of candidate videos based on theranking.
 15. The computer-implemented method of claim 12, wherein theplurality of viewer categories classify viewers based on at least one ofdemographic characteristics of viewers and online behavioralcharacteristics of viewers.
 16. The computer-implemented method of claim12, further comprising: for each respective viewer category among theplurality of viewer categories, assigning a respective quality score tothe respective viewer category; identifying a first watch time of thefirst video by a user in a first viewer category among the plurality ofviewer categories; in response to identifying the first watch time,increasing a performance score of a creator of the first video by anamount that is based on the first watch time and the respective qualityscore assigned to the first viewer category; identifying a second watchtime of the first video by a user in a second viewer category among theplurality of viewer categories; and in response to identifying thesecond watch time, increasing the performance score of the creator ofthe first video by an amount that is based on the second watch time andthe respective quality score assigned to the second viewer category. 17.The computer-implemented method of claim 16, comprising determining therespective quality scores assigned to respective viewer categories basedon revenue generated as a result of viewers in the respective viewercategories watching videos hosted by the computing system.
 18. Thecomputer-implemented method of claim 12, wherein, for each respectiveviewer category among the plurality of viewer categories, theaccumulated watch time for the respective viewer category indicates asum total of watch times of the first video by users within therespective viewer category.
 19. The computer-implemented method of claim12, wherein: serving the video content of the first video forpresentation at the first computing device comprises streaming the videocontent of the first video to the first computing device, and the firstcomputing device is configured to (i) receive the stream of the videocontent of the first video from the computing system and (ii) play thestream of the video content of the first video.
 20. One or morenon-transitory computer-readable media having instructions storedthereon that, when executed by one or more processors, cause performanceof operations comprising: obtaining, at a computing system, data thatindicates, for each respective computing device among a plurality ofcomputing devices to which at least a portion of a first video hosted bythe computing system was served, a respective watch time for the firstvideo that occurred at the respective computing device; for each of therespective watch times for the first video, attributing the respectivewatch time to one or more viewer categories among a plurality of viewercategories based on one or more characteristics of a user of therespective computing device at which the respective watch time occurred;for each respective viewer category among the plurality of viewercategories, accumulating the respective watch times that have beenattributed to the respective viewer category to generate an accumulatedwatch time for the respective viewer category, wherein the accumulatedwatch time for the respective viewer category indicates how long thefirst video was presented to users within the respective viewercategory; determining whether to provide video content of the firstvideo to a first user using a criterion that is based at least on theaccumulated watch time for the first video for a particular viewercategory among the plurality of viewer categories that matches the firstusers; and in response to the computing system determining that thecriterion for providing the video content of the first video to thefirst user is satisfied, serving, to a first computing device of thefirst user, the video content of the first video for presentation at thefirst computing device.
 21. The one or more non-transitorycomputer-readable media of claim 20, wherein determining whether toprovide video content of the first video to the first user comprisesdetermining whether the accumulated watch time of the first video forthe particular viewer category that matches the first user meets athreshold watch time.
 22. The one or more non-transitorycomputer-readable media of claim 20, wherein determining whether toprovide video content of the first video to the first user comprises:ranking a plurality of candidate videos, including the first video,based at least on respective accumulated watch times of the plurality ofcandidate videos for the particular viewer category that matches thefirst user; and selecting video content for one or more videos of theplurality of candidate videos based on the ranking.
 23. The one or morenon-transitory computer-readable media of claim 20, wherein: serving thevideo content of the first video for presentation at the first computingdevice comprises streaming the video content of the first video to thefirst computing device, and the first computing device is configured to(i) receive the stream of the video content of the first video from thecomputing system and (ii) play the stream of the video content of thefirst video.